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Project  Goals:  The  objective  of  this  proposal  wass  to  promote  the  use  of  computational  models  as  a 
tool  for  enhancing  our  understanding  of  human  social  dynamics.  This  proposal  facilitated  an 
interdisciplinary  collaboration  between  computer  science  and  social  science  research,  addressing  the 
relationship  between  emotion,  cognition  and  behavior,  specifically  focusing  on  computationally 
modeling  the  consequences  of  emotion  for  decision-making  in  social  contexts  and  the  role  of  culture  in 
moderating  these  effects.  The  effort  advanced  our  paradigm  of  model-driven  experimentation  (using 
computational  models  of  human  behavior  to  generate  predictions  that  can  be  validated  through  human 
experimentation).  The  findings  demonstrate  the  generality  of  the  approach  by  extending  prior  empirical 
findings  to  several  domains  of  human  social  interaction  and  assess  the  universality  of  proposed  cognitive 
models  by  contrasting  the  behavior  across  several  experimental  tasks  and  demographic  groups. 

Primary  Accomplishments  and  experimental  findings:  The  research  performed  as  part  of  this  grant 
served  to  advance  to  use  of  computational  models  of  human  emotional  processes  and  innovated  the 
use  of  such  models  to  study  human  social  emotional  processes.  Accomplishments  can  be  grouped  into 
three  interrelated  efforts.  First,  the  project  extended  and  validated  computational  models  of  intra¬ 
personal  processes:  specifically,  research  examined  how  an  individual's  emotions  are  impacted  by 
structure  of  tasks  using  appraisal  theories  of  emotions  as  a  theoretical  basis  for  both  model 
development  and  hypothesis  generation.  Second,  research  examined  the  inter-personal  emotional 
processes:  specifically,  how  emotional  signals  (e.g.,  facial  expressions)  generated  by  one  individual  might 
impact  the  beliefs,  feelings,  and  decisions  of  another.  This  work  pioneered  the  use  of  virtual 
confederates,  driven  by  the  computational  model,  to  unpack  interpersonal  emotional  processes.  Finally, 
a  third  thrust  examined  the  relationship  between  situational  appraisals  and  expressed  emotion: 
specifically,  studies  examined  if  there  was  a  stable  relationship  between  what  participants  actually 
showed  on  their  faces  and  manipulations  of  aspects  of  tasks  that  impact  participant's  appraisals.  This 
research  advanced  the  use  of  computer  vision  techniques  to  explore  psychological  problems. 

•  Intra-personal  processes:  This  work  primarily  focused  on  how  emotions  differentially  arise 
within  an  individual  based  on  the  structure  of  their  interactions  with  environment  and 
specifically  on  modeling  the  temporal  dynamics  of  emotion  processes.  These  were 


examined  through  a  series  of  experimental  tasks.  Studies  examined  how  structural  aspects 
of  a  situation,  predicted  by  appraisal  theory,  can  influence  emotional  responses  and 
subsequent  decision  making.  An  especial  focus  of  this  work  was  examining  how  different 
sequences  of  emotional  events  might  impact  people's  choice  of  different  strategies  to  cope 
with  emotion.  Work  followed  a  "model-driven  experimentation"  paradigm,  wherein  models 
were  used  to  simulate  situations  and  generate  specific  temporally  predictions  about  felt 
emotion  and  coping  responses  that  were  subsequently  tested  in  laboratory  experiments. 

The  work  is  described  in  detail  in  several  publications  by  Gratch  and  Marsella  and  by  Gratch 
and  Mao  (listed  below) 

•  Interpersonal  processes.  The  expression  of  emotion  can  influence  the  decisions  of  other 
social  partners  and  a  major  focus  of  the  grant  was  to  test  the  conjecture  that  appraisal 
theory  serves  as  an  effective  explanatory  framework  for  predicting  these  influences.  A  major 
and  novel  contribution  of  the  grant  was  the  development  of  reverse  appraisal  theory.  This 
theory  claims  that  people  interpret  expressions  of  emotion  in  the  context  of  co-occurring 
events  to  reverse-engineer  how  the  expressor  is  appraising  the  situation.  These  reverse- 
engineered  appraisals  then  provide  a  basis  for  inferring  the  other  party's  goals.  Thus, 
reverse-appraisal  theory  proposes  a  specific  mediation  model  for  how  people  make  mental- 
state  judgments  from  emotional  information  in  social  situations.  This  mediation  models  was 
tested  and  verified  through  a  series  of  studies  led  by  de  Melo  and  served  as  the  basis  of  his 
dissertation  on  this  topic.  The  details  are  described  in  the  publications  below 

•  Emotional  expression:  Appraisal  theory  argues  that  patterns  of  appraisals  are  reflected  in 
observable  behavior  such  as  facial  expressions  of  emotion.  As  part  of  the  goal  of  this  effort 
was  to  demonstrate  that  displays  of  emotion  driven  by  the  intra-personal  model  impact 
human  decisions  (i.e.,  reverse  appraisal),  this  work  sought  to  verify  that  emotional  displays 
seen  'in  nature'  actuall  reflect  appraisal  processes.  This  conjecture  was  examined  in  one  line 
of  research  that  placed  people  in  experimental  tasks  where  appraisals  could  be  manipulated 
and  facial  displays  measured  by  automatic  video-based  methods.  The  conjecture  was 
supported.  Details  can  be  found  in  the  article  by  Gratch,  Cheng,  Marsella  and  Boberg  listed 
below. 
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•  Research  Staff 

o  Jamison  Moore 
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Grant  related  publications: 

•  de  Melo,  Carnevale,  Gratch.  Humans  vs.  Computers:  The  Effect  of  Perceived  Agency  on 
People's  Decision  Making.  Under  review 

•  Rainer  Reisenzein,  Eva  Hudlicka,  Mehdi  Dastani,  Jonathan  Gratch,  Koen  Hindriks,  Emiliano 
Lorini,  and  John-Jules  Meyer.  Computational  Modeling  of  Emotion:  Towards  Improving  the 
Inter-  and  Intradisciplinary  Exchange.  IEEE  Transactions  on  Affective  Computing,  (in  press) 

•  Jonathan  Gratch,  Lin  Cheng,  Stacy  Marsella  and  Jill  Boberg.  Felt  emotion  and  social  context 
determine  the  intensity  of  smiles  in  a  competitive  video  game.  10th  IEEE  International 
Conference  on  Automatic  Face  and  Gesture  Recognition.  Shanghai,  China,  April  2013 

•  Wenji  Mao  and  Jonathan  Gratch.  Modeling  Social  Causality  and  Responsibility  Judgment  in 
Multi-Agent  Interactions:  Extended  abstract.  23rd  International  Joint  Conference  on  Artificial 
Inteligence.  Beijing,  China,  Aug,  2013 

•  de  Melo,  C.,  Gratch,  J.,  Carnevale,  P.  (2013). The  effect  of  agency  on  the  impact  of  emotion 
expressions  on  people's  decision  making.  Paper  to  be  presented  at  the  International 
Conference  of  Affective  Computing  and  Intelligent  Interaction  (ACM). 

•  de  Melo,  C.,  Carnevale,  P.,  &  Gratch,  J.  (2013).  People's  biased  decisions  to  trust  and 
cooperate  with  agents  that  express  emotions.  Paper  presented  at  the  T rust  Workshop  at  the 
Autonomous  Agents  and  Multiagent  Systems  (AAMAS)  Conference. 

•  de  Melo,  C.,  Carnevale,  P.,  &  Gratch,  J.  (2013).  Agent  or  avatar?  Using  virtual  confederates  in 
conflict  management  research.  Paper  to  be  presented  at  the  Annual  Meeting  of  the 
Academy  of  Management  (AOM). 

•  de  Melo,  C.,  Carnevale,  P.,  &  Gratch,  J.  (2013).  Reading  people's  minds  from  emotion 
expressions  in  social  decision  making.  Paper  to  be  presented  at  the  Annual  Meeting  of  the 
International  Association  for  Conflict  Management  (IACM). 


•  de  Melo,  C.,  Carnevale,  P.,  &  Gratch,  J.  (2012).  The  effect  of  virtual  agent's  emotion  displays 
and  appraisals  on  people's  decision  making  in  negotiation.  To  appear  in  Proceedings  of  The 
12th  International  Conference  on  Intelligent  Virtual  Agents  (IVA'12).  pdf 

•  de  Melo,  C.,  Carnevale,  P.,  Read,  S.,  &  Gratch,  J.  (2012).  Reverse  appraisal:  The  importance 
of  appraisals  for  the  effect  of  emotion  displays  on  people's  decision-making  in  a  social 
dilemma.  In  Proceedings  of  The  34th  Annual  Meeting  of  the  Cognitive  Science  Society 
(CogSci'12). 

•  de  Melo,  C.,  Carnevale,  P.,  Gratch,  J.  (2012).  The  impact  of  emotion  displays  in  embodied 
agents  on  emergence  of  cooperation  with  people.  Presence:  Teleoperators  and  Virtual 
Environments  Journal  20(5),  449-465. 

•  de  Melo,  C.,  Carnevale,  P.,  Read,  S.,  Antos,  D.,  &  Gratch,  J.  (2012).  Bayesian  model  of  the 
social  effects  of  emotion  in  decision-making  in  multiagent  systems.  To  appear  in 
Proceedings  of  Autonomous  Agents  and  Multiagent  Systems  (AAMAS'12). 

•  de  Melo,  C.  (2012).  The  interpersonal  effect  of  emotion  in  decision-making  and  social 
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Virtual  Worlds,  23,  3-4,  331-342. 

•  de  Melo,  C.,  Carnevale,  P.,  Read,  S.,&  Gratch,  J.  (2012).  Reverse  appraisal:  Appraisals 
mediate  the  effect  of  emotion  displays  in  decision-making  in  a  social  dilemma.  Poster 
presented  at  the  Emotion  SPSP  Pre-Conference  Meeting. 

•  Mao,  Wenji,  and  Jonathan  Gratch.  (2012).  "Modeling  social  causality  and  responsibility 
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273. 
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•  Keynote  speaker,  Workshop  on  Empathic  and  Emotional  Agents  at  the  International  Conference 
on  Autonomous  Agents  and  Multiagent  Systems,  Valencia,  Spain,  June  2012 

•  Invited  Panelist,  International  Conference  on  Cognitive  Modeling,  Berlin,  April  2012 

•  Invited  Speaker,  Northwestern  University  Cognitive  Science  Seminar  Series,  Chicago,  IL 
November  2011 

•  Invited  Panelist,  International  Conference  on  Affective  Computing  and  Intelligent  Interaction, 
Memphis,  TN,  October  2011 

•  Keynote  speaker,  Workshop  on  Standards  in  Emotion  Modeling,  Lorentz  Center,  Leiden.  August 
2011 

•  Keynote  speaker,  IEEE  International  Conference  on  Automatic  Face  and  Gesture  Recognition, 
Santa  Barbara,  March  2011 


Abstract 


The  objective  of  this  proposal  was  to  promote  the  use  of  computational  models  as  a  tool  for  enhancing 
understanding  of  human  social  dynamics,  specifically  the  relationship  between  emotion,  cognition  and 
behavior,  and  on  computationally  modeling  the  consequences  of  emotion  for  decision-making  in  social 
contexts.  Accomplishments  can  be  grouped  into  three  interrelated  efforts.  First,  the  project  extended  and 
validated  computational  models  of  intra-personal  processes:  specifically,  research  examined  how  an 
individual’s  emotions  are  impacted  by  structure  of  tasks  and  how  task  events  unfold  over  time.  Second, 
research  enhanced  fundamental  understanding  of  inter-personal  emotional  processes:  specifically,  how 
emotional  signals  (e.g.,  facial  expressions)  generated  by  one  individual  might  impact  the  beliefs,  feelings, 
and  decisions  of  another.  This  work  introduced  reverse  appraisal  theory  as  a  mechanism  to  explain  such 
social  effects.  Finally,  the  research  produced  insights  into  relationship  between  situational  appraisals  and 
expressed  emotion:  specifically,  studies  examined  if  there  was  a  stable  relationship  between  what 
participants  actually  showed  on  their  faces  and  manipulations  of  aspects  of  tasks  that  impact  participant’s 
appraisals.  This  project  led  to  one  PhD  dissertation. 


